Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix for systemd-resolved DNS incompatibility #82

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Fix for systemd-resolved DNS incompatibility #82

wants to merge 1 commit into from

Conversation

vannrt
Copy link
Contributor

@vannrt vannrt commented Dec 7, 2018 •
edited
Loading

This problem occurs because kube-dns on systems using systemd-resolved
copy 127.0.0.53 from the host's /etc/resolv.conf.

Since 127.0.0.53 is a loopback address, dns queries never get past
kube-dns causing our conformance tests to fail on DNS related issues.

More discussion here: kubernetes/kubernetes#45828

Related issues:
kubernetes/kubeadm#787
kubernetes/kubeadm#273
kubernetes/kubeadm#845

The upstream fix is now in v1.11.

Without the fix, the kubedns and dnsmasq containers would copy the host's /etc/resolv.conf:

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "systemd-resolve --status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
search platform9.sys

After the fix:

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 10.105.16.2
nameserver 10.105.16.4
search platform9.sys

@vannrt vannrt requested review from dlipovetsky, sarun87, puneetguptanitj and ojmhetar and removed request for dlipovetsky and sarun87 December 7, 2018 20:39
vannrt
vannrt commented Dec 7, 2018
View reviewed changes
constants/constants.go
@@ -75,7 +75,7 @@ const (
NodeadmKubeletSystemdDropinFilename = "20-nodeadm.conf"
NodeadmKubeletSystemdDropinTemplate = `[Service]
Environment="KUBELET_DNS_ARGS=--cluster-dns={{ .ClusterDNS }} --cluster-domain={{ .ClusterDomain }}"
Environment="KUBELET_EXTRA_ARGS=--max-pods={{ .MaxPods }} --fail-swap-on={{ .FailSwapOn }} --hostname-override={{ .HostnameOverride }} --kube-api-qps={{ .KubeAPIQPS }} --kube-api-burst={{ .KubeAPIBurst }} --feature-gates={{ .FeatureGates}} --eviction-hard={{ .EvictionHard }} --cpu-manager-policy={{ .CPUManagerPolicy }} --kube-reserved={{ .KubeReservedCPU }}"
Environment="KUBELET_EXTRA_ARGS=--resolv-conf=/run/systemd/resolve/resolv.conf --max-pods={{ .MaxPods }} --fail-swap-on={{ .FailSwapOn }} --hostname-override={{ .HostnameOverride }} --kube-api-qps={{ .KubeAPIQPS }} --kube-api-burst={{ .KubeAPIBurst }} --feature-gates={{ .FeatureGates}} --eviction-hard={{ .EvictionHard }} --cpu-manager-policy={{ .CPUManagerPolicy }} --kube-reserved={{ .KubeReservedCPU }}"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably ask one of our customers how they were able to bypass this issue. Based on last night's demo, they seem to prep the machine's networking prior to running cctl.

@vannrt
Fix for systemd-resolved DNS incompatibility
f48df82 
This problem occurs because kube-dns on systems using systemd-resolved
copy 127.0.0.53 from the host's /etc/resolv.conf.

Since 127.0.0.53 is a loopback address, dns queries never get past
kube-dns causing our conformance tests to fail on DNS related issues.

More discussion here: kubernetes/kubernetes#45828

Related issues:
kubernetes/kubeadm#787
kubernetes/kubeadm#273
kubernetes/kubeadm#845

The upstream fix is now in v1.11.

Without the fix, the kubedns and dnsmasq containers would copy the host's `/etc/resolv.conf`:
```
\# This file is managed by man:systemd-resolved(8). Do not edit.
\#
\# This is a dynamic resolv.conf file for connecting local clients to the
\# internal DNS stub resolver of systemd-resolved. This file lists all
\# configured search domains.
\#
\# Run "systemd-resolve --status" to see details about the uplink DNS servers
\# currently in use.
\#
\# Third party programs must not access this file directly, but only through the
\# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
\# replace this symlink by a static file or a different symlink.
\#
\# See man:systemd-resolved.service(8) for details about the supported modes of
\# operation for /etc/resolv.conf.

nameserver 127.0.0.53
search platform9.sys
```

After the fix:
```
\# This file is managed by man:systemd-resolved(8). Do not edit.
\#
\# This is a dynamic resolv.conf file for connecting local clients directly to
\# all known uplink DNS servers. This file lists all configured search domains.
\#
\# Third party programs must not access this file directly, but only through the
\# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
\# replace this symlink by a static file or a different symlink.
\#
\# See man:systemd-resolved.service(8) for details about the supported modes of
\# operation for /etc/resolv.conf.

nameserver 10.105.16.2
nameserver 10.105.16.4
search platform9.sys
```
@vannrt vannrt force-pushed the private/vteves/resolv-conf branch from 3b39821 to f48df82 Compare December 7, 2018 20:45
ojmhetar
ojmhetar reviewed Dec 11, 2018
View reviewed changes
constants/constants.go
@@ -75,7 +75,7 @@ const (
NodeadmKubeletSystemdDropinFilename = "20-nodeadm.conf"
NodeadmKubeletSystemdDropinTemplate = `[Service]
Environment="KUBELET_DNS_ARGS=--cluster-dns={{ .ClusterDNS }} --cluster-domain={{ .ClusterDomain }}"
Environment="KUBELET_EXTRA_ARGS=--max-pods={{ .MaxPods }} --fail-swap-on={{ .FailSwapOn }} --hostname-override={{ .HostnameOverride }} --kube-api-qps={{ .KubeAPIQPS }} --kube-api-burst={{ .KubeAPIBurst }} --feature-gates={{ .FeatureGates}} --eviction-hard={{ .EvictionHard }} --cpu-manager-policy={{ .CPUManagerPolicy }} --kube-reserved={{ .KubeReservedCPU }}"
Environment="KUBELET_EXTRA_ARGS=--resolv-conf=/run/systemd/resolve/resolv.conf --max-pods={{ .MaxPods }} --fail-swap-on={{ .FailSwapOn }} --hostname-override={{ .HostnameOverride }} --kube-api-qps={{ .KubeAPIQPS }} --kube-api-burst={{ .KubeAPIBurst }} --feature-gates={{ .FeatureGates}} --eviction-hard={{ .EvictionHard }} --cpu-manager-policy={{ .CPUManagerPolicy }} --kube-reserved={{ .KubeReservedCPU }}"
Copy link
Contributor

@ojmhetar ojmhetar Dec 11, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though this is a temp workaround before 1.11, I think /run/systemd/solve/resolv.conf should be placed into a constant and substituted via this template like the other flags.

@sarun87
Copy link
Contributor

sarun87 commented Jan 9, 2019

@vannrt As I understand it,1.11 will not require this fix. Do we still want to address this as a patch for 1.10.11?

@dlipovetsky dlipovetsky mentioned this pull request Jan 29, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ojmhetar ojmhetar ojmhetar left review comments

@puneetguptanitj puneetguptanitj Awaiting requested review from puneetguptanitj

@dlipovetsky dlipovetsky Awaiting requested review from dlipovetsky

@sarun87 sarun87 Awaiting requested review from sarun87

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@vannrt @sarun87 @ojmhetar